Move pipes external metadata processing to its own layer. #29311
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Apr 15, 2025
This was referenced Apr 16, 2025
Closed
Closed
dpeng817
force-pushed
the
airlift_jobs_ui
branch
from
7c45b11 to
1c6bdac
Compare
dpeng817
force-pushed
the
dpeng817/airlift_external_metadata
branch
from
99a0cfe to
c5a373e
Compare
dpeng817
force-pushed
the
airlift_jobs_ui
branch
from
1c6bdac to
620bee6
Compare
dpeng817
force-pushed
the
dpeng817/airlift_external_metadata
branch
from
c5a373e to
82c84a6
Compare
dpeng817
force-pushed
the
airlift_jobs_ui
branch
from
620bee6 to
9c048e4
Compare
dpeng817
force-pushed
the
dpeng817/airlift_external_metadata
branch
from
82c84a6 to
858028b
Compare
dpeng817
force-pushed
the
airlift_jobs_ui
branch
from
9c048e4 to
cc88eb2
Compare
dpeng817
force-pushed
the
dpeng817/airlift_external_metadata
branch
from
858028b to
56f34b8
Compare
dpeng817
force-pushed
the
airlift_jobs_ui
branch
from
cc88eb2 to
5431920
Compare
dpeng817
force-pushed
the
dpeng817/airlift_external_metadata
branch
from
56f34b8 to
b342d25
Compare
dpeng817
force-pushed
the
airlift_jobs_ui
branch
from
5431920 to
3fdc27c
Compare
dpeng817
force-pushed
the
dpeng817/airlift_external_metadata
branch
from
b342d25 to
c314fbe
Compare
dpeng817
force-pushed
the
airlift_jobs_ui
branch
from
3fdc27c to
05cae98
Compare
dpeng817
force-pushed
the
dpeng817/airlift_external_metadata
branch
from
c314fbe to
30c200c
Compare
dpeng817
force-pushed
the
airlift_jobs_ui
branch
from
05cae98 to
bfa843c
Compare
dpeng817
force-pushed
the
dpeng817/airlift_external_metadata
branch
from
30c200c to
e8ab560
Compare
dpeng817
force-pushed
the
airlift_jobs_ui
branch
from
bfa843c to
71b6fb3
Compare
dpeng817
force-pushed
the
dpeng817/airlift_external_metadata
branch
from
e8ab560 to
9b2413d
Compare
dpeng817
force-pushed
the
airlift_jobs_ui
branch
from
71b6fb3 to
bcc288a
Compare
dpeng817
force-pushed
the
dpeng817/airlift_external_metadata
branch
from
9b2413d to
697b8df
Compare
dpeng817
force-pushed
the
airlift_jobs_ui
branch
from
bcc288a to
2b05e81
Compare
dpeng817
force-pushed
the
dpeng817/airlift_external_metadata
branch
from
697b8df to
397a404
Compare
dpeng817
force-pushed
the
airlift_jobs_ui
branch
from
2b05e81 to
4201f3a
Compare
dpeng817
force-pushed
the
dpeng817/airlift_external_metadata
branch
from
397a404 to
3bec9a9
Compare
dpeng817
force-pushed
the
airlift_jobs_ui
branch
from
4201f3a to
0efd739
Compare
dpeng817
force-pushed
the
dpeng817/airlift_external_metadata
branch
from
3bec9a9 to
562d549
Compare
dpeng817
force-pushed
the
airlift_jobs_ui
branch
from
0efd739 to
b7d73ec
Compare
dpeng817
force-pushed
the
dpeng817/airlift_external_metadata
branch
from
562d549 to
431dc53
Compare
dpeng817
force-pushed
the
airlift_jobs_ui
branch
from
b7d73ec to
11356b9
Compare
dpeng817
force-pushed
the
dpeng817/airlift_external_metadata
branch
from
431dc53 to
55a3c16
Compare
This was referenced Apr 25, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Pipes and airlift have kinda the same mandate here to use a well-structured format for defining metadata in an external process, that may not have access to the dagster package. So we can just move the processing code to a shared place accessible to both, and make the language more generic to instead be "external" metadata.
How I Tested These Changes
Existing pipes metadata tests